An improved polarization recording approach to reduce speckle noise in digital holography is proposed. Multiple off-axis holograms are obtained by rotating the linear polarization state of both illumination and reference wave simultaneously. By averaging the intensity fields, the speckle noise in the reconstructed images is well suppressed. Statistical evaluation of the experimental results shows the effectiveness and improvement of the proposed method.